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Abstract 


The  purpose  of  this  research  was  to  demonstrate  a  methodology  using  an  Epoch- 
Era  Analysis  to  quantify  and  estimate  the  value  of  design  flexibility  early  in  the 
Department  of  Defense’s  (DOD)  acquisition  life  cycle.  This  method  was  implemented 
using  a  possible  replacement  to  the  Air  Force’s  fighter-trainer  aircraft  as  a  baseline  and  a 
set  of  future  requirements  that  would  change  the  baseline.  An  existing  Cost  Estimating 
Relationship  tool  was  utilized  in  conjunction  with  a  decision  tree  modeling  approach  to 
accommodate  uncertain  future  needs.  Sensitivity  analysis  was  performed  to  identify 
model  parameters  with  dominant  effects  on  the  recommended  design  strategies.  The 
results  indicated  that  this  methodology  can  quantitatively  measure  design  flexibility  using 
existing  tools  when  key  assumptions  are  made.  The  methodology  exists  as  a  proof  of 
concept  within  the  domain  of  aircraft  to  quantitatively  measure  design  flexibility  early  in 
the  acquisition  life  cycle.  Further  research  is  required  to  characterize  the  assumptions  of 
this  study  and  to  test  this  methodology  in  other  domains  to  validate  its  broader 
applicability. 
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EXPLORING  A  METHOD  TO  QUANTITATIVELY  MEASURE  DESIGN 
FLEXIBILITY  EARLY  IN  THE  DEFENSE  ACQUISITION  LIFE  CYCLE 


I.  Introduction 


General  Issue 

In  the  world  of  DOD  acquisition,  flexibility  is  often  touted  as  a  valuable  “ility.” 
However,  effectively  designing  flexibility  into  systems  and  objectively  measuring  the 
outcome  is  exceedingly  difficult.  The  added  costs  of  designing  flexibility  early  in  the 
acquisition  life  cycle,  as  well  as  the  future  costs  incurred,  can  be  difficult  to  justify 
without  a  means  of  valuing  said  flexibility. 

A  developing  example  is  the  replacement  aircraft  for  the  Northrop  T-38C  Talon. 
It  is  the  current  airframe  used  for  the  fighter/bomber  track  of  the  United  States  Air 
Force’s  Specialized  Undergraduate  Pilot  Training  (SUPT).  First  introduced  in  1961,  the 
T-38  received  a  host  of  upgrades  over  its  life  cycle  to  maintain  the  trainer’s  relevance  to 
newer  generations  of  fighters  and  bombers  culminating  in  the  latest  version,  the  T-38C 
[United  States  Air  Force  2014].  While  the  T-38C  has  undergone  a  service  life  extension 
program,  the  Flight  Training  System  Program  Office  located  at  Wright-Patterson  Air 
Force  Base  projects  the  T-38C  airframe  will  reach  the  end  of  its  useful  life  in  2020 
[United  States  Air  Force  2014], 

The  McDonnell  Douglas  T-45  Goshawk  is  the  current  airframe  used  by  the  Navy 
as  its  aircraft  carrier-capable  jet  trainer.  First  introduced  in  1988,  the  T-45  received  a 
glass  cockpit  upgrade  and  other  modernizations  for  continued  use  as  a  contemporary  jet 
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trainer  [United  States  Navy  2009].  Similar  to  the  T-38C,  the  T-45  will  likely  see 
continued  use  as  the  Navy’s  jet  trainer  through  continuous  modernization  efforts. 

With  the  expiration  of  the  T  -38C  airframe  approaching  in  2020  and  the  time- 
consuming  nature  of  large  acquisitions  programs,  the  T-X  FoS  (Family  of  Systems), 
Advanced  Pilot  Trainer  (APT)  acquisitions  process  began  in  the  fall  of  2003  [United 
States  Air  Force  2014].  The  T-45  Goshawk  will  also  reach  the  end  of  its  useful  life, 
perhaps  ten  to  fifteen  years  after  the  T-38C.  If  there  were  a  method  to  quantitatively 
capture  the  value  of  designing  flexibility  into  the  T-38C  replacement  to  accommodate  the 
Navy’s  T-45  replacement,  decision-makers  could  be  better  informed  on  whether  or  not 
the  additional  resources  to  design  flexibility  into  the  T-38C  replacement  would  yield  an 
acceptable  return  on  investment.  Similarly,  additional  future  requirements  could  impact 
and  possibly  be  accommodated  by  designing  flexibility  into  the  T-38C  replacement 
trainer. 

Problem  Statement 

Given  the  DOD’s  budget-constrained  environment,  there  is  further  pressure  to 
investigate  methods  to  reduce  life  cycle  cost.  The  Analysis  of  Alternatives  completed  by 
the  T-X  FoS  ATP  program  offered  a  range  of  materiel  solutions  differing  in  the 
performance  capabilities  of  the  airframe  being  acquired  relevant  to  the  Air  Force’s  jet 
trainer  requirements.  Rather  than  focus  solely  on  the  Air  Force’s  fighter/bomber 
requirements,  the  concept  of  design  flexibility  and  its  quantitative  measure  was  explored 
in  order  to  accommodate  other  user’s  requirements.  For  this  study,  three  additional 
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requirements  were  considered:  Navy  trainers,  Special  Operations  trainers,  and  Heavy 
airframe  trainers. 

Research  Obj  ectives/Questions/Hypotheses 

The  objective  of  this  research  was  to  demonstrate  a  method  that  models  the  life 
cycle  cost  (LCC)  impacts  associated  with  engineering  design  flexibility  into  a  system 
early  in  the  acquisition  process.  The  T-X  program  served  as  a  demonstration  of  this 
method  which  attempted  to  characterize  the  cost  of  adding  flexibility  to  the  baseline 
design  and  its  impact  (or  lack  thereof)  on  the  LCC  of  the  modified  system. 

Research  Focus 

There  are  a  multitude  of  factors  that  can  affect  the  LCC  of  an  airframe.  The  scope 
of  this  research  focused  on  how  design  changes  driven  by  uncertain  requirements  in  the 
early  phases  of  acquisitions  affected  the  LCC  of  the  proposed  airframes.  LCC  includes 
the  following  costs:  research  and  development,  investment,  operating  and  support,  and 
disposal  [Defense  Acquisition  University  2013].  In  this  research,  the  calculation  of  life 
cycle  costs  were  built  into  the  cost  estimating  relationship  (CER)  tool  provided  by  the  Air 
Force  Life  Cycle  Management  Center  (AFLCMC/XZE)  and  divided  into  development, 
production,  operations  and  support  (O&S),  and  disposal  costs. 

Investigative  Questions 

1 .  How  can  design  flexibility,  as  a  proxy  measure  for  design  flexibility,  be 
quantifiably  measured  in  the  early  stages  of  development  of  a  system? 

2.  Can  we  measure  the  impact  to  expected  FCC  stemming  from  design  changes 
to  accommodate  flexibility  given  uncertain  future  requirements? 
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3.  Can  a  general  method  to  quantify  the  value  of  design  flexibility  be  developed 
and  applied  to  other  domains  beyond  airframes? 

Methodology 

A  literature  review  examining  the  existing  work  on  how  to  define  flexibility  and 
methods  to  measure  flexibility  was  conducted.  Based  upon  the  information  found,  a 
definition  and  metric  for  design  flexibility  was  established  and  utilized  for  this  study. 

An  Epoch-Era  Analysis  approach  was  used  to  define  discrete  manifestations  of 
the  proposed  system  and  evaluate  the  differences  in  LCC  [Ross  2006].  By  utilizing 
existing  cost  estimation  relationship  models  developed  by  AFLCMC/XZE,  separate 
epochs  were  created  and  examined  to  study  the  effects  of  design  flexibility  on  the  LCC  of 
a  proposed  replacement  to  the  T-38C  trainer  aircraft. 

The  baseline  system  was  one  of  many  proposed  replacements  to  the  T-38C  trainer 
aircraft  that  met  Air  Force  trainer  requirements.  Epochs  were  added  to  the  baseline 
system  by  including  three  additional  uncertain  future  requirements:  Navy,  Special 
Operations,  and  Heavy.  Specific  design  changes  to  the  baseline  Air  Force  trainer 
requirements  were  considered  to  capture  the  requirements  of  the  uncertain  future 
requirements.  Based  on  notional  probabilities  of  occurrence,  the  expected  LCCs  of  each 
epoch/era  were  compared  to  detennine  how  the  differing  epoch  variables  impacted  each 
era. 

The  cost  of  design  flexibility  was  compared  to  the  cost  of  building  separate 
discrete  system  that  met  the  possible  requirements  of  the  Navy,  Special  Operations,  and 
Heavy  airframes.  A  comparison  between  designing  for  flexibility  and  developing 
separate  discrete  system  shed  light  onto  the  value  of  early  design  for  flexibility. 
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The  current  SUPT  syllabus,  the  Initial  Capabilities  Document  of  the  T-X 
program,  and  subject  matter  experts  were  utilized  to  create  a  rough  baseline  of  Air  Force 
requirements.  The  cost  estimates  developed  for  this  research  do  not  represent  actual 
program  estimates  in  order  to  allow  the  open  distribution  of  the  results.  Separate  baseline 
epochs  adding  requirements  were  created  to  examine  the  impact  that  design  flexibility 
had  on  LCC.  The  additional  capability  required  by  each  epoch  variable  is  notional  and 
the  assumptions  made  are  discussed  in  their  appropriate  sections. 

The  separate  epochs  are  associated  with  variables  that  assumed  a  range  of  values 
that  distinguished  one  from  another.  The  ranges  of  values  were  captured  in  the  CER  tool 
and  the  outputs  were  recorded.  This  data  served  as  inputs  into  a  decision  tree  that 
calculated  expected  LCC  for  a  wide  variety  of  possible  outcomes.  For  further  insight, 
sensitivity  analyses  were  conducted  on  several  inputs  to  detennine  their  impacts  on  the 
tradespace  of  expected  LCC. 

Assumptions/Limitations 

Because  the  purpose  of  this  study  was  to  demonstrate  a  method,  actual  values  for 
variable  inputs  and  model  outputs  were  not  necessarily  accurate  compared  to  real-world 
values.  Rather,  they  attempted  to  capture  a  range  of  reasonable  values  and  suggest  trends 
associated  with  design  flexibility. 

The  cost  model  used  in  this  study  was  limited  by  the  manner  of  its  inputs.  This 
study  worked  around  these  input  limitations  which  were  noted  in  their  appropriate 
sections.  Other  assumptions  were  noted  as  necessary  in  this  research  paper. 
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Implications 

This  exploratory  model  to  quantify  design  flexibility  was  first  created  in  the 
context  of  the  T-X  program.  However,  the  model  should  be  broad  enough  to 
accommodate  other  domains.  With  a  general  method  to  help  quantify  design  flexibility, 
decision-makers  at  all  levels  could  benefit  from  increased  insight  into  adding,  removing, 
or  avoiding  additional  requirements.  In  the  long  run,  the  intent  is  to  reduce  total  costs 
associated  with  acquiring  new  systems  and  modifying  existing  systems  by  providing  a 
better  understanding  of  the  returns  on  investment  inherent  in  design  flexibility. 
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II.  Literature  Review 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  establish  a  theoretical  framework,  define  key 
tenns,  and  identify  studies  and  models  that  supported  the  modeling  of  design  flexibility. 

Literature 

Multi-Attribute  Tradespace  Exploration  with  Concurrent  Design  (MATE-CON) 
MATE-CON  is  a  powerful  tool  to  evaluate  multiple  architectures  and  their 
respective  designs  [Ross  2003],  For  longer  studies,  the  design-level  analysis  can  be  used 
to  re-evaluate  the  architecture-level  analysis  to  further  improve  the  accuracy  of  the 
models  used  and  the  architectures  selected. 

MATE-CON  can  be  broken  down  into  five  phases:  need  identification, 
architecture  solution  exploration,  architecture  evaluation,  design  solution  exploration,  and 
design  evaluation.  Developed  and  used  extensively  by  Dr.  Adam  Ross  at  the 
Massachusetts  Institute  of  Technology  (MIT),  he  describes  the  process  as  such: 


“The  Need  Identification  phase  motivates  the  entire  project,  providing  the  needs, 
mission,  and  scope  for  the  project.  MATE-CON  is  the  marriage  of  the 
architecture-level  exploration  and  evaluation  (MATE)  with  the  design-level 
exploration  and  evaluation  (CON).  Architecture-level  exploration  and  evaluation 
is  accomplished  using  models  and  simulations  to  transform  a  large  set  of  design 
vectors  to  attributes  and  then  evaluating  each  set  of  attributes  in  utility-cost  space. 
The  set  of  modeled  design  vectors,  or  architectures,  are  analyzed  in  utility-cost 
space  and  the  best  architectures  are  selected  for  the  design-level  exploration  and 
evaluation”  [Ross  2003,  70]. 
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With  MATE-CON,  Ross  introduced  a  straightforward  and  powerful  tool  to 
examine  a  large  set  of  designs  and  their  impact  to  the  value  on  the  system  known  as  an 
Epoch  and  Era  Analysis  (EEA). 

Epoch-Era  Analysis 

EEA  is  an  approach  developed  alongside  MATE-CON  that  models  several 
proposed  designs  and  compares  cost  and  utility  metrics.  EEA  was  originally  created  to 
be  used  with  MATE;  the  original  implementation  modeled  a  vast  number  of  space  system 
designs  and  measures  their  respective  cost  and  utilities  [Ross,  et  al.  2004], 

An  Epoch  “is  a  time  period  that  bounds  the  change  scenario  during  which  utility 
functions,  constraints,  design  concepts,  available  technologies,  and  articulated  attributes 
are  defined”  [Ross  2006,  170].  Similar  to  economics  analysis,  EEA  seeks  to  break  down 
a  complex  problem  into  a  series  of  simple  problems.  For  both  short  and  long  run 
analyses,  many  system  attributes  and  constraints  are  “fixed  in  the  short  run  (Epoch),  but 
variable  in  the  long  run  (Era)”  [Ross  2006,  170-171]. 

Each  epoch  has  an  identified  beginning  state  and  ending  state.  Each  epoch  has 
key  variables  that  impact  the  defined  value  of  the  system  that  differentiates  it  from  other 
epochs  [Fitzgerald,  Ross  and  Rhodes  2011],  A  meaningful  epoch  variable  also  captures 
the  uncertainty  associated  with  the  respective  epoch.  When  multiple  epochs  are  ordered 
together,  an  era  that  highlights  a  potential  progression  of  system  states  over  a  period  of 
time  is  created  [Ross  2006].  The  era  allows  an  analyst  to  measure  how  changing  epoch 
variables  affect  the  system  over  a  period  of  time  based  upon  a  metric  of  the  analyst’s 
choosing. 
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Defining  and  Measuring  Flexibility 

It  is  important  to  distinguish  two  types  of  flexibility,  process  and  design: 

“The  literature  on  flexibility  in  engineering  design  addresses  two  distinct  problems: 
the  first  one  focuses  on  the  flexibility  of  the  design  process,  and  the  second  one  on 
the  flexibility  of  the  design  itself  (not  the  process  through  which  a  product  or  a 
system  is  designed).  This  distinction  between  the  flexibility  of  the  process  and 
flexibility  of  the  design  is  not  often  made  in  the  literature,  and  it  sometimes  adds  to 
the  confusion”  [Saleh,  Mark,  and  Jordan  2009,  313], 

According  to  the  distinction  between  the  two  types  of  flexibility,  this  research 
focused  on  a  method  to  model  design  flexibility.  Within  the  academic  community,  there 
are  some  inconsistencies  that  exist  in  the  definition  of  design  flexibility.  A  literature 
survey  was  conducted  by  Ryan  examining  2 1  varying  definitions  of  design  flexibility 
spanning  1997-2010.  The  differences  among  the  definitions  included:  does  the  system 
actually  change,  does  the  change  happen  quickly  or  cost-effectively,  is  the  change 
foreseeable,  does  the  change  occur  before  or  after  fielding  [Ryan,  Jacques  and  Colombi 
2013].  Many  of  these  definitions  attempt  to  capture  value  but  this  is  accomplished  in 
slightly  different  ways. 

Adaptability  is  defined  as  the  ability  of  a  system  that  can  modify  its  capabilities 
without  external  intervention  [Ryan,  Jacques  and  Colombi  2013].  Machine  learning  is  a 
prime  example  of  adaptability  where  an  adaptable  system  can  improve  on  its  existing 
capabilities  without  the  need  for  additional  programming.  Commonality  seeks  to  create 
value  by  reducing  unique  parts  requirements  and  establishing  economies  of  scale  with 
producing  and  maintaining  shared  parts  [Simpson  and  D’Souza  2008].  With 
commonality,  the  certainty  of  additional  requirements  is  implied  with  the  effort  to 
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standardize  parts  between  multiple  systems.  The  F-35  Joint  Strike  Fighter  is  an  example 
where  commonality  was  implemented  as  a  cost  reduction  measure. 

Of  particular  interest  is  design  flexibility,  which  is  defined  as  a  system  designed 
with  certain  characteristics  that  may  not  optimize  the  immediate  set  of  requirements,  but 
easily  allows  the  system  to  accommodate,  via  modifications,  new  requirements  after  the 
system  has  been  fielded  [Saleh,  Mark,  and  Jordan  2009].  Design  flexibility  is  also 
described  as  “the  measure  of  how  easily  a  system’s  capabilities  can  be  modified  in 
response  to  external  change”  [Ryan,  Jacques  and  Colombi  2013].  In  this  definition, 
“easily”  refers  to  the  cost  effectiveness  and  timely  manner  of  the  modification  to  the 
system. 

In  the  literature,  the  definitions  of  design  flexibility  and  design  robustness  are 
similar  and  often  confused  with  one  another,  but  are  distinctly  different.  Design 
flexibility  “implies  an  ability  to  satisfy  changing  requirements  by  changing  the  system 
after  the  system  has  been  fielded”  and  “an  ability  of  the  design  to  be  changed  in  order  to 
track  requirements  changes”  [Saleh,  Mark,  and  Jordan  2009,  316].  Saleh  used  a 
spacecraft  example  to  demonstrate  his  definition  of  design  flexibility.  He  states  that  the 
spacecraft  may  require  new  functionalities  as  events  and/or  new  data  become  available. 
The  changing  functionality  is  also  quite  likely  due  to  the  design  lifetime  of  most 
spacecraft,  which  demands  that  the  spacecraft  incorporate  design  flexibility  to 
accommodate  these  future  changes  [Saleh,  Mark,  and  Jordan  2009].  This  differs  from 
design  robustness  which  captures  the  ability  of  a  design  to  “satisfy  a  fixed  set  of 
requirements,  despite  changes  in  the  environment  or  within  the  system”  [Saleh,  Mark, 
and  Jordan  2009,  316]. 
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After  defining  flexibility,  the  next  challenge  is  how  to  measure  flexibility. 
Currently,  the  DOD’s  LCC  includes  research  and  development,  investment,  operating 
and  support,  and  disposal  costs  over  a  system’s  life  cycle  [Defense  Acquisition 
University  2013].  In  addition  to  these  costs,  LCC  estimates  take  into  consideration 
potential  program  risks  [Defense  Acquisition  University  2013],  Although 
comprehensive,  DOD  LCC  estimates  are  ultimately  a  static  measure  assuming  the 
program  will  not  deviate  from  the  acquisition  program  baseline  [Defense  Acquisition 
University  2013].  The  DOD’s  method  of  estimating  LCC  does  not  attempt  to  measure 
flexibility  in  any  way. 

Ryan  proposed  a  methodology  that  measured  a  cost  coined  as  Current  Expected 
Value  Life  Cycle  Cost  Curve  (CEVLCCC)  [E.  Ryan,  et  al.  2013].  This  methodology 
sought  to  capture  cost  impacts  due  to  potential  changes  to  the  baseline.  Of  interest,  this 
methodology  accommodated  the  ability  for  the  baseline  system  to  respond  to  potential 
changes.  If  a  system  could  respond  cost-effectively  to  a  change  to  its  baseline  it  would 
drive  down  the  associated  cost  penalty  applied  to  that  system  in  order  to  achieve  design 
flexibility  [E.  Ryan,  et  al.  2013],  Ryan’s  methodology  required  that  each  system  design 
candidate  be  of  sufficient  maturity  that  traditional  life  cycle  cost  estimates  could  be 
implemented.  Unfortunately,  this  is  not  the  case  for  this  study.  However,  Ryan’s 
approach  to  identify  CEVLCCC  as  one  proxy  metric  to  capture  value  was  used  in  a 
similar  manner  for  this  study. 

Cost  Estimating  Tool 

Engineers  at  AFLCMC/XZE  have  developed  an  Excel-based  airframe  LCC 
estimating  tool  that  takes  user  inputs  depending  on  the  function  and  design  features  of  the 
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hypothetical  aircraft  and  outputs  LCC  as  a  function  of  development,  production,  O&S, 
and  disposal  costs.  Calculations  are  taken  from  past  data  with  input  from  subject  matter 
experts. 

An  example  of  one  input  is  the  type  of  aircraft  being  developed.  The  options 
within  the  tool  allow  for  tighter,  bomber,  cargo,  and  trainer  aircraft.  Based  upon  this 
selection,  the  tool  draws  upon  information  from  past  airframes  of  that  type  to  estimate 
several  costs  which  include,  but  are  not  limited  to:  annual  operation  costs,  maintenance 
costs,  and  development  costs. 

Although  a  powerful  tool,  the  estimating  tool  has  limitations  as  well.  Unless  a 
specific  component  is  coded  into  the  tool,  the  only  method  to  account  for  an  additional 
feature  is  to  incorporate  it  into  an  existing  input.  For  example,  if  a  user  wanted  another 
estimate  of  an  airframe  with  a  larger  landing  gear,  no  explicit  input  for  landing  gear  type 
exists.  The  way  around  this  limitation  is  to  adjust  the  total  empty  aircraft  weight  to 
accommodate  a  reasonable  value  for  the  modification.  Typically,  only  major  components 
such  as  engines  and  avionics  have  dedicated  input  values  [AFLCMC/XZE  2013]. 
Conclusion 

There  are  many  models  and  techniques  that  improve  the  analysis  of  multiple 
alternatives,  define  flexibility,  measure  flexibility,  and  compare  flexibility.  This  research 
intends  to  take  these  models  and  incorporate  them  into  a  unified  method  that  better 
predicts  and  analyzes  the  cost  associated  with  design  flexibility  using  the  T-X  program  as 
a  source  of  data.  Although  based  off  data  from  the  T-X  program,  the  steps  to  implement 
this  method  should  be  applicable  to  other  disciplines  as  well. 
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ITT.  Methodology 


Chapter  Overview 

The  purpose  of  this  chapter  was  to  describe  the  research  methodology  of  this 
study,  explain  the  sample  selection,  describe  the  procedures  used  in  designing  the 
instrument  and  collecting  the  data,  and  provide  an  explanation  of  the  procedures  used  to 
analyze  the  data. 

Based  on  the  literature  review,  design  flexibility  was  measured  based  upon  the 
ability  for  an  airframe  in  development  to  accommodate  future  potential  requirements. 

The  metric  used  to  measure  design  flexibility  was  the  expected  LCC,  a  proxy  measure  of 
design  flexibility  [Ryan,  Jacques  and  Colombi  2013]. 

A  quantitative  and  predictive  research  methodology  was  used  for  this  study. 
Epochs  and  their  associated  variables  were  established  based  upon  the  four  different 
aircraft  requirements:  Air  Force,  Navy,  Special  Operations,  and  Heavy  (transport).  Next, 
the  cost  estimating  relationship  (CER)  tool  developed  by  AFLCMC/XZE  was  used  to 
generate  LCC  estimates  based  on  the  different  aircraft  design  requirements.  A  separate 
LCC  estimate  was  generated  to  compare  the  cost  of  an  Air  Force  system  with  additional 
requirements  incorporated  versus  the  cost  of  a  new  dedicated  system.  To  examine  impact 
the  additional  requirement  had  on  the  system,  Ross’  epoch  and  era  analysis  was  used  to 
observe  the  differences  and  trends  across  all  the  eras.  In  addition  to  the  existing  tools, 
general  assumptions  towards  the  requirements  of  different  users  were  made  to  model  the 
different  aircraft  configurations  based  on  the  intended  user.  Multiple  sensitivity  analyses 
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were  conducted  to  examine  the  impact  of  uncertain  parameters  on  the  output  of  the 
method. 

Although  modeling  past  behavior  and  trends  is  important  for  a  decision-maker, 
future  behavior  of  a  system  is  more  important  than  explaining  past  observations  [Shmueli 
2007].  Ultimately,  the  method  developed  in  this  study  should  be  of  use  to  decision¬ 
makers  attempting  to  quantify  the  cost  associated  with  designing  flexibility  into  their 
systems  regardless  of  the  cost  models  used. 

Sample 

Reiterating  the  previous  point  on  Ross’  MATE-CON,  the  process  can  be  broken 
down  into  five  phases:  need  identification,  architecture  solution  exploration,  architecture 
evaluation,  design  solution  exploration,  and  design  evaluation  [Ross  2003].  At  the  time 
of  this  study,  the  Air  Force  had  already  accomplished  the  need  identification,  architecture 
solution  exploration,  and  architecture  evaluation  for  the  T-X  program.  The  need 
identification  requirement  was  established  first  as  the  T-X  FoS  ATP’s  Initial  Capabilities 
Document  and  later  as  the  Capability  Development  Document.  Both  architecture 
solution  exploration  and  evaluation  were  completed  as  identified  by  the  “family  of 
systems”  approach  to  acquisitions  [Ross,  et  al.  2004],  The  Air  Force  expanded  its 
acquisition  focus  beyond  just  the  airframe  and  recognized  the  importance  of  capturing  all 
aspects  of  a  new  trainer. 

This  study  continues  with  design  solution  exploration  and  design  evaluation.  The 
design  solution  exploration  considered  the  impact  of  adding  three  additional  requirements 
to  the  replacement;  a  Navy,  Special  Operations,  and  Heavy  (transport)  requirements.  The 
design  evaluation  was  the  result  of  this  study. 
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Testing 

The  first  step  in  examining  the  impact  of  additional  requirements  to  a  T-X 
baseline  design  was  to  establish  separate  epochs.  The  Air  Force  epoch  costs  reflected  the 
baseline  design  while  the  additional  epochs  captured  the  cost  to  add  additional 
requirements.  Ideally,  the  additional  epochs  would  capture  only  their  respective 
requirements  rather  than  include  the  AF  requirements.  But  due  to  the  limitation  of  the 
CER  tool  and  the  notional  nature  of  the  additional  requirements,  this  was  a  necessary 
assumption.  The  epoch  variables  represented  the  future  possibility  that  the  requirements 
and/or  the  design  parameters  demanded  of  the  baseline  system  could  change. 

For  this  study,  the  epoch  variables  and  their  possible  range  of  values  represented 
realizations  of  requirements  that  were  not  originally  identified  in  the  Acquisition  Program 
Baseline  (APB).  However,  the  inclusion  of  the  epoch  variables  was  assumed  to  occur 
before  production  costs  were  incurred.  This  point  is  further  clarified  in  Chapter  4. 

It  is  important  to  note  that  AFLCMC/XZE  developed  multiple  baseline  Air  Force 
only  requirement  (AF)  epochs  for  their  own  estimating  purposes.  This  study  utilized  their 
estimate  of  a  single-engine  supersonic  aircraft  as  the  baseline  AF  estimate.  The  different 
epoch  variables  were  notional  and  chosen  based  on  discussions  with  subject  matter 
experts  at  AFLCMC/XZE  as  well  as  the  modeling  limitations  of  the  CER  tool.  This  was 
deemed  reasonable  as  the  purpose  of  the  model  was  to  explore  a  new  method  to  measure 
design  flexibility  and  the  accuracy  of  the  epoch  variables  was  deemed  less  important  for 
this  initial  demonstration  of  the  methodology.  Table  1 :  CER  Tool  Assumptions, 
summarizes  the  assumptions  related  to  the  CER  tool  that  were  made  for  this  study. 

These  assumptions  were  reiterated  as  appropriate  in  the  latter  sections. 
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Table  1:  CER  Tool  Assumptions 


_ CER  Tool  Assumptions _ 

Disposal  costs  were  omitted  from  LCC  calculation _ 

Additional  epoch  requirements  were  added  to  the  baseline  AF  estimate 

N  and  H  epoch  variables  were  converted  to  aircraft  weight _ 

Timeline  for  development,  production,  and  retirement  remained  constant 
Annual  production  of  aircraft  remained  constant _ 


Separate  epochs  capturing  Navy  requirements  (N),  Special  Operations 
requirements  (SO),  and  Heavy  requirements  (H)  were  created  based  upon  the  identified 
epoch  variables.  Ideally,  unique  requirements  for  each  epoch  would  be  used  to  capture 
their  respective  LCCs.  However,  AF  specific  requirements  were  the  only  requirements 
available.  Therefore,  the  N,  SO,  and  H  requirements  were  additions  to  the  AF  baseline 
and  assumed  to  fully  encompass  AF  requirements.  Sensitivity  analysis  helped  address 
the  unrealistic  assumption  that  N,  SO,  and  H  epochs  fully  encompassed  AF  requirements. 

In  regard  to  the  number-of-engines  variable,  rather  than  an  addition  to  the  AF 
baseline,  the  number  of  engines  represented  a  potential  change  to  the  AF  baseline  and 
was  implemented  as  such  in  this  study.  In  hindsight,  a  more  appropriate  approach  could 
characterize  the  number  of  engines  as  a  design  decision  rather  than  a  stochastic  event. 
Additional  epochs  and  eras  could  be  created  to  capture  the  decision  to  incorporate  one  or 
two  engines.  To  scope  the  possible  number  of  eras  to  a  reasonable  total,  the  number  of 
engines  was  treated  as  a  stochastic  event  and  the  costs  associated  with  a  one-or-two 
engine  design  were  averaged  into  the  appropriate  epoch  costs.  A  summary  of  the  epochs 
and  their  variables  are  listed  in  Table  2:  Epoch  Summary. 
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Table  2:  Epoch  Summary 


Epoch 

Epoch  Variables 

Air  Force  Requirement 
(AF) 

N/A 

Navy  Requirement  (N) 

Number  of  Engines 
Reinforced  Landing  Gear 

Tail  Hook 

#  of  Combat-Coded  A/C 

Special  Operations  (SO) 

Improved  Avionics 
#  of  Combat-Coded  A/C 

Heavy  Requirements  (H) 

Number  of  Engines 

Cockpit  Interface  Interchangeability 
#  of  Combat-Coded  A/C 

The  N  epoch  variables  are  meant  to  represent  features  similar  to  the  existing  T-45 
Goshawk.  Two  features  included  a  tail  hook  and  a  stronger  structure  supporting  the 
landing  gears  to  accommodate  carrier  landings  [United  States  Navy  2009],  The  addition 
of  these  requirements  was  reflected  in  the  CER  tool  as  additional  aircraft  weight.  The 
option  for  one  or  two  engines  was  also  a  potential  requirement  due  to  the  Navy’s  affinity 
for  two  engines  as  seen  on  their  current  F-18  and  past  F-14  fighter  aircraft.  This  option 
was  explicitly  specified  in  the  CER  tool.  The  purpose  of  the  epoch  variables  and  their 
combinations  with  a  one  or  two  engine  design  was  to  generate  discrete  LCC  points  in 
order  to  calculate  the  mean  LCC  of  the  epoch.  It  was  assumed  the  range  of  LCC  across 
the  range  of  epoch  variables  was  a  uniform  distribution  in  the  absence  of  specific 
distribution  data.  This  assumption  was  extended  to  the  SO  and  H  epochs  as  well.  The 
method  could  easily  accommodate  other  LCC  distributions  such  as  triangle  and  nonnal. 

The  SO  epoch  variable  of  improved  avionics  captured  the  clandestine  nature  of 
special  operations.  This  can  be  manifested  as  avionics  that  allow  for  low-level  flying  at 


night  or  a  sophisticated  communications/electronics  package  similar  to  those  equipped  in 
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the  EC- 130  that  support  special  operations  [United  States  Air  Force  2005],  The  avionics 
weight  was  an  explicit  input  in  the  CER  tool  that  was  utilized  to  capture  this  SO 
requirement. 

The  H  epoch  translated  cockpit  interface  interchangeability  into  aircraft  weight  in 
the  CER  tool.  Without  an  explicit  input  in  the  CER  tool  to  capture  this  epoch  variable, 
aircraft  weight  was  chosen  as  a  proxy  input  that  best  represented  the  addition  of  cockpit 
interface  interchangeability.  This  epoch  variable  was  meant  to  capture  the  ability  for  the 
AF  aircraft  to  accommodate  a  different  cockpit  layout  more  appropriate  for  a  heavy 
trainer  aircraft  vs  a  fighter-bomber  trainer  aircraft.  Notionally,  this  could  include  location 
of  throttle,  joystick  type,  joystick  position,  and  instrument  layout.  Like  the  N  epoch,  a 
one-or-two  engine  design  was  combined  with  each  possible  weight  configuration  to 
calculate  the  mean  LCC  of  the  H  epoch. 

For  the  N,  H,  and  SO  epochs,  the  number  of  combat  coded  aircraft  was  another 
epoch  variable  they  shared  and  was  directly  input  into  the  CER  tool.  This  variable 
captured  the  uncertainty  in  fleet  size  representative  of  any  aircraft  acquisition.  Along 
with  the  other  epoch  specific  variables,  the  number  of  combat-coded  aircraft  was  input 
with  every  combination  of  aircraft  weight,  avionics  weight,  and  number  of  engines  as 
appropriate  to  create  a  range  of  LCCs  and  ultimately  calculate  the  average  LCC  of  the 
epoch. 
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Eight  eras  were  evaluated  in  this  study: 

1 .  AF  =  AF  epoch  only 

2.  AFN  =  AF  and  N  epochs 

3.  AFSO  =  AF  and  SO  epochs 

4.  AFH  =  AF  and  H  epochs 

5.  AFNSO  =  AF,  N,  and  SO  epochs 

6.  AFNH  =  AF,  N,  and  H  epochs 

7.  AFSOH  =  AF,  SO,  and  H  epochs 

8.  AFNSOH  =  AF,  N,  SO,  and  H  epochs 

The  eras  evaluated  in  this  study  were  not  collectively  exhaustive  as  a  whole,  but 
collectively  exhaustive  of  the  eras  that  included  AF  epochs.  Each  era  represented  a 
possible  future  reality.  A  design  strategy  represented  the  decision-maker’s  choice  to 
pursue  or  forgo  design  flexibility.  Among  the  possible  design  strategies  involving  the  AF 
epoch,  the  AFNSO  and  AFSOH  eras  were  omitted  from  the  analysis. 

The  design  strategies  evaluated  in  this  study  were: 

1 .  AF  =  AF  epoch  only 

2.  AFN  =  AF  and  N  epochs 

3.  AFSO  =  AF  and  SO  epochs 

4.  AFH  =  AF  and  H  epochs 

5.  AFNH  =  AF,  N,  and  H  epochs 

6.  AFNSOH  =  AF,  N,  SO,  and  H  epochs 

The  two  omitted  design  strategies  were  not  considered  because  the  six  design 
strategies  selected  were  representative  of  multiple  epoch  eras.  The  additional  design 
strategies  would  not  demonstrate  any  additional  insight  to  the  methodology  than  those 
chosen  to  be  evaluated. 

Once  a  design  strategy  was  selected,  each  of  the  eight  eras  became  a  possible 
realization.  Palisade’s  Precision  Tree  add-in  for  Excel  was  used  to  visualize  and 
calculate  expected  FCCs  based  upon  probabilities  of  an  era  occurring.  Figure  1 :  Partial 
Decision  Tree  illustrates  a  portion  of  the  decision  tree  used  in  this  study.  The  green 
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square  represents  a  decision  node  for  the  possible  choices  that  a  decision-maker  could 
make.  In  this  study,  the  decision  node  represents  aircraft  design  strategies  that  could  be 
pursued.  The  design  strategy  is  the  decision-maker’s  choice  to  design  flexibility  (or  not) 
into  the  baseline  AF  epoch.  Red  circles  represent  chance  nodes  with  branches  that 
capture  a  set  of  mutually  exclusive  and  collectively  exhaustive  outcomes. 


-  Expected  LCC 


Design  Strategy 

$90,403 


■  AFN 


FALSE, 


■AFN 


■  AFSO 


■  AFH 


10.0% 


Realization 

$92,429 


AFNH 


AFSOH 


■  AFNSOH 


$53,113 


30.0% 


$85,460 

15.0% 


$77,801 

15.0% 


S90,145 


10.0% 

9.0% 


S122,493 

6.0% 


$114,834 

5.0% 


$147,181 


◄ 

◄ 

◄ 

◄ 

◄ 

◄ 


Figure  1:  Partial  Decision  Tree 


The  probability  associated  with  each  era  represent  the  probability  that  the  era  will 
be  realized  for  a  given  design  strategy.  Because  the  eight  eras  including  an  AF  epoch  are 
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mutually  exclusive  and  collectively  exhaustive,  the  sum  of  the  probabilities  must  equal 
100%.  It’s  feasible  that  the  probabilities  among  the  possible  eras  could  change 
depending  on  the  design  strategy  chosen.  However,  with  no  good  method  to  capture  how 
a  chosen  design  strategy  would  impact  the  probabilities  of  its  outcomes,  it  was  assumed 
that  the  probabilities  of  each  outcome  across  all  design  strategies  were  equal.  The 
probabilities  of  the  eight  eras  are  listed  in  Table  3:  Era  Summary. 


Table  3:  Era  Summary 


Era 

Epochs 

Probability  of  Occurring 

AF 

AF 

10% 

AFN 

AF  +  N 

30% 

AFSO 

AF  +  SO 

15% 

AFH 

AF  +  H 

15% 

AFNSO 

AF  +  N  +  SO 

10% 

AFNH 

AF  +  N  +  H 

9% 

AFSOH 

AF  +  SO  +  H 

6% 

AFNSOH 

AF  +  N  +  SO  +  H 

5% 

The  probability  of  occurrence  for  each  era  was  determined  based  upon  the 
foreseeable  need  for  a  new  trainer  aircraft.  Due  to  the  introduction  of  5th  generation 
fighter  aircraft  and  improvements  in  avionics,  modern  day  trainers  must  prepare  pilots  to 
become  familiar  with  these  improved  capabilities.  The  T-45  Navy  trainer  entered  service 
in  1991  and  primarily  prepared  Navy  aviators  for  the  F-18  Hornet  [United  States  Navy 
2009].  With  the  introduction  of  the  F-35  as  the  Air  Force  and  Navy’s  next  fighter,  it  was 
deemed  that  the  eras  that  captured  Navy  requirements  had  a  higher  probability  of 
occurrence  than  the  other  comparable  eras.  Given  that  the  probability  of  one  era 
occurring  impacted  the  probabilities  of  the  remaining  eras,  the  probabilities  were  treated 


21 


as  dependent.  Due  to  the  dependent  nature  of  the  probabilities,  there  are  conditional 
probabilities  that  are  implied  by  the  probability  distribution  among  the  possible  eras. 

The  trainer  aircraft  used  to  prepare  SUPT  pilots  for  many  of  the  platforms  that 
support  special  operations  entered  service  in  the  70’s  [United  States  Navy  2012],  The 
aircraft  used  to  train  SUPT  pilots  for  Heavy  missions  was  adopted  in  1992  but  has  not 
seen  significant  upgrades  since  its  adoption  [United  States  Air  Force  2005].  Based  upon 
the  age  of  the  trainer  aircraft  used  to  satisfy  the  Special  Operations  and  Heavy 
requirements,  these  eras  were  assigned  the  next  highest  probability  of  occurrence.  The 
AF  only  era  was  considered  the  next  probable  era  followed  by  AFNSOH  due  to  the  small 
chance  that  all  additional  requirements  would  be  realized. 

The  additional  requirements  of  each  epoch  variable  in  this  study,  other  than  the 
number  of  combat-coded  aircraft,  were  meant  to  be  design  changes  to  the  existing  Air 
Force  baseline.  For  example,  the  N  epoch  variable  of  a  tail  hook  would  be  a  flexible 
design  addition  to  the  Air  Force  baseline.  This  design  consideration  would  require  some 
modification  to  the  Air  Force  design  to  cost  effectively  accommodate  a  tail  hook  in  the 
future  if  the  need  arose.  This  design  flexibility  approach  was  used  in  contrast  to  simply 
adding  the  full  design  requirement  of  a  tail  hook  to  the  AF  baseline  because  there  was  no 
certainty  that  an  era  such  as  AFN  would  be  realized.  Also,  the  additional  requirements 
levied  on  the  baseline  AF  system  would  have  negative  impacts  on  the  cost  and 
performance  of  the  system. 

Designing  flexibility  into  any  system  comes  at  a  cost.  To  calculate  the  impact  to 
LCC  of  the  additional  requirements  to  the  baseline  design,  a  subjective  impact  value  from 
0.0  -  1.0,  was  added  to  the  production  and  operation  and  sustainment  (O&S)  cost  of  the 
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relevant  era.  The  subjective  impact  was  a  penalty  that  design  flexibility  imposed  on  the 
AF  baseline  that  resulted  in  higher  production  and  O&S  costs.  In  regards  to  development 
costs,  the  difference  in  additional  development  cost  to  accommodate  flexible  design  was 
added  to  the  AF  development  cost.  The  nature  of  the  epoch  variables  was  consistent  with 
Saleh’s  definition  of  design  flexibility  that  a  system  could  be  modified  more  easily  if 
additional  requirements  were  levied  on  the  baseline  system  [Saleh,  Mark,  and  Jordan 
2009]. 

The  values  of  each  epoch  variable  were  limited  by  the  CER  tool.  Explicit  design 
changes  could  not  be  directly  added  as  inputs  into  the  CER  tool  for  N  and  H 
requirements.  These  requirements  were  translated  into  aircraft  weight,  an  input  the  CER 
tool  could  accommodate.  The  breakdown  of  the  epoch  variables,  their  baseline  values, 
and  their  possible  values  are  listed  in  Table  4:  Epoch  Variables  below. 
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Table  4:  Epoch  Variables 


Epoch  Variables 

Baseline  Value 

Possible  Values 

Number  of  Engines 

1  Engine 

1  Engine 

2  Engines 

Reinforced  Landing 
Gear 

A/C  Weight 
9900  lbs 
(+0  lbs) 

Additional  A/C  Weight 

200  lbs 

300  lbs 

400  lbs 

500  lbs 

600  lbs 

700  lbs 

800  lbs 

Tail  Hook 

A/C  Weight 
9900  lbs 
(+0  lbs) 

Additional  A/C  Weight 

400  lbs 

500  lbs 

1300  lbs 

1400  lbs 

Avionics 

Avionics  Weight 
360  lbs 
(+0  lbs) 

Additional  Avionics  Weight 
100  lbs 

200  lbs 

600  lbs 

700  lbs 

Cockpit  Interface 
Interchangeability 

A/C  Weight 
9900  lbs 
(+0  lbs) 

Additional  A/C  Weight 

100  lbs 

200  lbs 

300  lbs 

400  lbs 

#  of  Combat  Coded  A/C 

350  A/C 

#  of  A/C 

N:  200,  250,  300 

SO:  150,200,250 

H:  150,200,250,300 

The  range  of  values  for  each  epoch  variable  was  notionally  selected.  Certain 
considerations  were  made  to  ensure  that  the  maximum  epoch  variable  values  remained 
reasonable.  For  example,  the  addition  of  a  reinforced  landing  gear  and  tail  hook  should 
not  increase  total  aircraft  weight  by  50%.  Similar  considerations  were  taken  to  detennine 
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the  minimum  values  for  each  epoch  variable.  Landing  gear  and  tail  hook  considerations 
were  deemed  to  have  a  larger  impact  to  aircraft  weight  than  the  ability  for  the  cockpit  to 
accommodate  separate  configurations.  The  values  for  the  number  of  combat-coded 
aircraft  were  chosen  based  on  the  total  number  of  trainer  aircraft  produced  for  each 
respective  epoch. 

Each  epoch  must  have  a  beginning  and  ending  state  identified.  Clearly,  the 
beginning  and  ending  state  of  the  AF  baseline  was  before  LCC  was  incurred  as  only  eras 
involving  the  AF  epoch  were  considered.  Any  additional  epochs  began  after 
development  costs  were  incurred  but  prior  to  production  costs.  The  decision  to  pursue  a 
design  strategy  must  be  made  prior  to  DOD  acquisition  Milestone  A.  The  described 
timeline  is  illustrated  in  Figure  2:  Era  Timeline  Assumption  using  the  AFN  design 
strategy  and  AFN  era  occurrence  as  an  example. 


Choose  AFN 

AF  Req 

Design  Strategy 

Decision  to 
Exercise  N 
Req'ts 


Incur AFN 

Incur AFN 

Dev  Cost 

Prod  Cost 

Incur  AFN 
O&S  Cost 


Time 


Milestone  A 


Figure  2:  Era  Timeline  Assumption 
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Once  the  AFN  design  strategy  is  chosen,  AFN  development  costs  are  incurred 
regardless  of  N  epoch  occurring  or  not  occurring.  The  decision  whether  to  exercise  the  N 
epoch  must  occur  after  development  cost  is  incurred  but  before  production  cost  is  fully 
incurred.  For  example,  if  the  decision  to  not  exercise  N  epoch  was  made  before 
development  costs  were  incurred,  then  no  further  investments  to  the  AF  baseline  should 
be  made  and  therefore,  no  additional  costs  to  the  AF  baseline  would  be  incurred.  Eras 
with  three  or  more  epochs  would  look  similar  to  Figure  2:  Era  Timeline  Assumption  with 
the  chosen  design  strategy  occurring  before  Milestone  A  and  the  decision  to  exercise  the 
additional  epochs  occurring  between  development  and  production  costs. 

If  the  decision  to  exercise  an  epoch  were  to  occur  after  production  costs  were 
incurred,  another  “penalty”  factor  would  be  required  to  account  for  the  extra  cost  of 
restarting  production  or  extending  the  production  timeline  beyond  original  estimates. 

This  is  illustrated  in  Figure  3  using  AFN  as  an  example. 


Choose  AFN 

AF  Req 

Design  Strategy 

Decision  to 
Exercise  N 
Req'ts 


Incur  AFN 

Incur  AF 

Dev  Cost 

Prod  Cost 

Incur  AFN 
O&S  Cost 


Time 


Milestone  A 


Incur  N 
Prod  Cost 


Figure  3:  Possible  Era  Realization 
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Other  penalties  such  as  loss  of  expertise,  additional  staff  support  costs,  lack  of 
bulk  orders  are  some  of  the  additional  costs  that  would  be  incurred.  This  research 
assumed  a  timeline  as  shown  in  Figure  2:  Era  Timeline  Assumption  due  to  the  inability  of 
the  CER  tool  to  calculate  the  aforementioned  penalty  to  LCC  and  avoid  unnecessary 
speculation. 

Another  necessary  assumption  was  once  a  design  strategy  was  chosen,  there  are 
no  “off  ramps”  available.  For  example,  after  AFN’s  development  cost  occurs  and  if  the 
decision  to  not  exercise  N  epoch’s  requirements  was  made,  there  was  no  option  that 
allowed  the  acquisition  to  revert  back  to  a  baseline  AF  design  and  produce  AF  aircraft, 
effectively  avoiding  any  future  cost  penalties  associated  with  design  flexibility. 

The  expected  cost  of  flexibility  was  measured  by  comparing  a  given  era’s 
expected  LCC  against  the  expected  LCC  of  the  era’s  respective  epochs.  The  expected 
LCC  of  AFN  was  measured  against  the  sum  of  the  LCC  of  AF  and  N  epochs  to  detennine 
if  there  was  cost  savings  associated  with  design  flexibility.  All  cost  figures  were 
calculated  in  Base  Year  2013  (BY2013)  dollars. 

Finally,  one  and  two-way  sensitivity  analyses  were  conducted  in  order  to  analyze 
how  changes  in  some  of  the  major  inputs  would  affect  expected  LCC.  Specifically,  era 
LCC  estimates,  probability  of  occurrence,  and  subjective  impact  sensitivity  analyses  were 
conducted.  Due  to  the  greater  uncertainty  associated  with  subjective  inputs,  a  sensitivity 
analysis  was  deemed  necessary  to  establish  trends  associated  with  those  inputs.  The  goal 
was  to  view  the  impact  that  the  subjective  inputs  had  on  the  output. 
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Summary 

The  goal  of  this  study  is  to  demonstrate  a  method  to  quantitatively  measure  the 
value  of  design  flexibility.  Reasonable  assumptions  were  made  to  overcome  limited 
information,  CER  tool  functionality,  and  to  properly  scope  this  research.  Other  notional 
assumptions  such  as  probability  of  occurrence  and  subjective  impact  were  the  subject  of 
sensitivity  analyses  to  identify  the  trending  impact  these  inputs  had  on  the  method’s 
output. 
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IV.  Analysis  and  Results 


Chapter  Overview 

This  chapter  covers  the  implementation  of  the  methodology  and  the  associated 
outcomes.  The  goal  of  this  analysis  was  to  demonstrate  the  method  proposed  in  Chapter 
3  and  to  quantify  the  cost  of  design  flexibility  in  the  context  of  the  T-X  acquisition 
scenario. 

Analysis 

The  first  step  of  analysis  was  to  generate  the  proper  data  points  and  establish  cost 
averages  for  each  epoch.  Using  the  CER  tool  provided  by  AFLCMC/XZE,  estimates  of 
development,  production,  and  O&S  costs  were  recorded  across  the  range  of  values 
established  by  the  epoch  variables.  The  disposal  cost  was  another  output  of  the  CER 
tool;  however,  disposal  costs  were  less  than  a  tenth  of  a  percent  of  the  total  LCC  for  each 
epoch.  Therefore,  disposal  costs  were  omitted  from  this  analysis.  Using  H  as  an 
example,  for  200  aircraft,  four  cost  estimates  were  generated  adding  100  lb  increments  to 
the  AF  baseline  and  four  additional  cost  estimates  were  generated  using  a  two  engine 
design.  This  process  was  repeated  twice  more  at  fleet  sizes  of  250  and  300  aircraft.  The 
average  across  all  the  H  cost  estimates  was  recorded  with  an  assumed  uniform 
distribution.  The  averages  of  each  epoch  are  summarized  in  Table  5:  Epoch  Cost 
Averages. 
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Table  5:  Epoch  Cost  Averages 


Epoch 

LCC 

Total  Dev 
Cost 

Unit 

Prod 

Cost 

Unit  O&S 
Cost 

#of 

A/C 

AF 

$48,236 

$2,916 

$39.72 

$89.76 

350 

N 

$37,625 

$2,949 

$48.36 

$90.02 

250 

SO 

$30,592 

$2,994 

$48.86 

$89.13 

200 

H 

$37,033 

$2,805 

$46.71 

$89.89 

250 

NOTE:  All  $  1 

igures  BY  13  in  millions 

These  averages  were  based  upon  inputs  in  the  CER  tool  that  estimated 
development  beginning  in  2016,  production  beginning  in  2020,  and  aircraft  retirement  in 
2047.  The  total  estimated  program  timeline  from  development  to  retirement  was  32 
years.  This  notional  timeline  was  based  upon  the  baseline  AF  estimate  provided  by 
AFLCMC/XZE. 

Based  upon  the  generation  of  this  data,  several  observations  were  made. 
Development  cost  was  independent  of  fleet  size.  Regardless  of  how  many  aircraft  were 
input  into  the  CER  tool,  development  costs  would  hold  constant.  As  indicated  in  SO’s 
unit  production  cost  average,  avionics  weight  played  a  larger  factor  in  production  due  to  a 
high  unit  production  cost.  Likewise,  SO’s  unit  O&S  cost  was  slighter  lower  than  the 
larger  fleet  epochs  due  to  avionics  weight  having  very  little  impact  on  O&S  which 
overcame  the  diseconomies  of  scale  associated  with  a  smaller  fleet  size.  This  was 
compared  to  N  and  H  where  additional  aircraft  weight  had  a  lesser  impact  on  production 
cost  than  avionics  weight,  but  a  slightly  larger  negative  impact  to  unit  O&S  cost 
compared  to  the  AF  baseline. 
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As  discussed  in  Chapter  3,  cost  estimates  for  each  epoch  were  generated  by 
adding  or  changing  requirements  to  the  AF  baseline.  Average  development  cost  for  H 
was  lower  than  the  AF  development  cost.  Although  initially  odd,  closer  examination 
revealed  that  while  all  other  inputs  were  held  constant  in  the  CER  tool,  switching  from  a 
one  engine  design  to  a  two  engine  design  would  decrease  development  costs  by 
approximately  $300  million.  According  to  the  CER  tool,  the  one  and  two  engine  designs 
are  modifications  to  existing  engines.  A  possible  explanation  for  the  reduced 
development  costs  for  a  two  engine  design  could  be  due  to  an  existing  two  engine  design 
that  is  closer  to  the  modified  engine  requirements  than  the  one  engine  design.  Because 
the  range  of  additional  weight  for  H  was  relatively  small  (100  lbs  -  400  lbs),  the  cost  to 
develop  the  additional  weight  did  not  exceed  the  decreased  development  cost  when 
averaged  across  all  possible  H  cost  estimates.  A  two  engine  design  was  not  without  its 
own  penalties.  Although  development  costs  associated  with  a  two  engine  design  would 
decrease  development  cost,  an  increase  in  the  production  cost  was  incurred. 

Before  calculating  expected  LCCs,  key  terms  were  defined.  Flexible  investment 
cost  was  the  cost  to  design  flexibility  into  the  AF  baseline  design  and  was  determined  by 
the  design  strategy  chosen  by  the  decision-maker.  Flexible  investment  cost  impacted 
the  baseline  development,  production,  and  O&S  costs,  and  was  incurred  regardless  of 
whether  the  option  to  implement  the  additional  capabilities  was  exercised.  The  flexible 
investment  development  cost  was  equal  to  the  AF  baseline  development  cost  plus  the 
additional  development  cost  to  accommodate  a  given  era’s  requirements.  The  exception 
to  this  was  AFH  where  the  development  cost  was  set  equal  to  AF  rather  than  using  a 
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development  cost  lower  than  the  baseline.  The  two  epoch  era  equations  listed  used  Navy 
requirements  as  an  example  in  Equation  1 . 

AFN  Flexible  Investment  Development  Cost  =  AFN  Dev  Cost  (  1  ) 

The  total  of  AF  unit  production  cost  plus  the  product  of  N  unit  production  cost 
and  N  subjective  impact  was  multiplied  by  the  number  of  AF  aircraft  to  calculate  the 
AFN  flexible  investment  production  cost.  The  subjective  impact  captured  the  additional 
cost  associated  with  producing  AF  aircraft  with  design  flexibility  that  could 
accommodate  a  potential  future  N  requirement  is  demonstrated  in  Equation  2. 

AFN  Flexible  Investment  Production  Cost  = 

#  of  AF  A/C  *  ((AF  unit  Prod  Cost)  + 

(N  unit  Prod  Cost  *  N  subjective  impact))  (  2  ) 

Figure  4:  Flexible  Investment  Production  Cost,  illustrates  Equation  2  where  the 
baseline  unit  production  cost  was  more  expensive  when  considering  the  design  flexibility 
required  of  the  potential  future  requirement  of  N.  The  unit  production  cost  impact  due  to 
flexible  design  was  a  small  percentage  of  the  unit  production  cost  of  N. 
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AF  Baseline 


AF  unit  Prod  Cost 

AFN  Flex  Inv  Prod  Cost 

AF  unit  Prod  Cost 

X 

X  =  (N  unit  Prod  Cost  *  N  subjective  impact) 

Figure  4:  Flexible  Investment  Production  Cost 

The  subjective  impact  was  a  notional  value  between  0-1.0  that  represented  how 
design  flexibility  negatively  affected  the  cost  of  producing  each  AF  aircraft.  A  value  of 
zero  indicated  no  impact  to  AF  production  costs.  The  purpose  of  the  subjective  impact 
was  to  model  the  penalty  of  design  flexibility  on  the  baseline  AF  system.  The  subjective 
impact  captured  the  design  change  to  AF  that  allowed  the  aircraft  to  be  easily  modified  to 
accommodate  future  epoch  requirements  if  the  need  arose. 

Notional  subjective  impact  values  are  listed  in  Table  6:  Subjective  Impacts.  The 
Navy  epoch  was  considered  to  have  the  greatest  cost  impact  to  unit  production  costs  due 
to  the  nature  of  accommodating  a  reinforced  landing  gear  and  a  tail  hook.  This  could 
result  in  a  heavier  frame  compared  to  the  AF  baseline  which  would  translate  into 
increased  production  and  O&S  costs.  The  special  operations  epoch  was  considered  to 
have  a  moderate  negative  impact  due  to  the  considerations  of  improved  avionics. 
Although  a  stronger  frame  than  the  AF  baseline  may  not  be  necessary,  the  space 
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necessary  to  accommodate  larger  avionics  could  have  a  moderate  impact  on  the  baseline 
design.  Finally,  the  Heavy  epoch  was  considered  to  have  the  least  negative  impact  as 
supporting  a  separate  control  scheme  within  the  cockpit  might  require  relatively  minimal 
changes  between  the  interfaces  of  the  cockpit  and  airframe. 


Table  6:  Subjective  Impacts 


Epoch 

Subjective  Impact 

N 

0.10 

SO 

0.07 

H 

0.05 

Flexible  investment  O&S  cost  was  treated  in  the  same  manner  as  flexible 
investment  production  cost.  The  AF  baseline  O&S  cost  incurred  a  penalty  for  the 
additional  flexibility  requirements  that  must  be  accommodated  throughout  the  life  cycle 
of  the  now  modified  AF  system  as  shown  in  Equation  3. 

AFN  Flexible  Investment  O&S  Cost  =  #  of  AF  A/C  *  ((AF  unit  O&S  Cost)  + 

(N  unit  O&S  Cost  *  N  subjective  impact))  (  3  ) 

The  flexible  investment  cost  of  the  extended  eras  AFNH  and  AFNSOH  followed 
the  same  convention  as  their  shorter  counterparts  with  a  small  exception  to  AFNSOH’s 
flexible  investment  production  cost  as  demonstrated  in  Equations  4-5.  The  flexible 
investment  development  cost  of  AFNSOH  aggregated  the  development  cost  delta  of  the 
additional  epochs.  As  previously  mentioned,  the  negative  cost  delta  between  AF  and  H 
development  cost  was  ignored. 

AFNSOH  Flexible  Investment  Development  Cost  =  AFN  Dev  Cost  + 

(AFSO  Dev  Cost  -  AF  Dev  Cost)  (  4  ) 
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AFNSOH  Flexible  Investment  Production  Cost  =  #  of  AF  A/C  * 

((AF  unit  Prod  Cost)  +  (N  unit  Prod  Cost  *  N  subjective  impact)  +  0.9  * 

[(SO  unit  Prod  Cost  *  SO  subjective  impact)  +  (H  unit  Prod  Cost  * 

H  subjective  impact)])  (  5  ) 

The  two  epoch  eras  have  assumed  an  additive  nature  of  accommodating  the 
flexible  design  of  additional  requirements.  The  general  convention  for  flexible 
investment  costs  for  AFNSOH  was  modified  by  the  inclusion  of  multiplying  0.9  to  the 
sum  of  the  flexible  design  impact  of  SO  and  H.  This  was  designed  to  capture  some  of  the 
non-additive  properties  of  producing  flexible  design  changes  to  roughly  the  same  area  of 
the  AF  aircraft.  It  can  be  argued  that  production  efficiencies  could  exist  when  modifying 
AF’s  baseline  avionics  as  demanded  by  SO’s  requirements  and  cockpit  interchangeability 
as  demanded  by  H’s  requirements.  A  relatively  high  value  of  0.9  was  chosen  to  capture  a 
minor  efficiency  because  the  avionics  in  an  A/C  are  not  necessarily  limited  to  the  general 
proximity  of  the  cockpit.  The  further  the  distance  flexible  design  work  occurs  away  from 
the  cockpit,  the  less  production  efficiencies  would  be  realized  between  SO  and  H 
requirements.  The  non-additive  attribute  exists  only  in  the  AFNSOH  and  AFSOH 
flexible  investment  production  cost  and  is  unique  to  these  two  eras. 

AFNSOH  Flexible  Investment  O&S  Cost  =  #  of  AF  A/C  *  ((AF  unit  O&S  Cost)  + 
(N  unit  O&S  Cost  *  N  subjective  impact)  +  (SO  unit  O&S  Cost  * 

SO  subjective  impact)  +  (H  unit  O&S  Cost  *  H  subjective  impact))  (  6  ) 

The  AFNH  era  arguably  did  not  realize  the  same  production  efficiencies  as 
AFNSOH  and  AFSOH  due  to  the  separate  areas  of  the  AF  aircraft  being  modified  as 
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demanded  by  N  and  H  requirements.  Therefore,  both  AFNH  flexible  investment 
production  and  O&S  costs  followed  similar  convention  to  the  shorter  eras  as  shown  in 
Equations  7-8. 

AFNH  Flexible  Investment  Production  Cost  =  #  of  AF  A/C  * 

((AF  unit  Prod  Cost)  +  (N  unit  Prod  Cost  *  N  subjective  impact)  + 

(H  unit  Prod  Cost  *  H  subjective  impact))  (  7  ) 


AFNH  Flexible  Investment  O&S  Cost  =  #  of  AF  A/C  *  ((AF  unit  O&S  Cost)  + 

(N  unit  O&S  Cost  *  N  subjective  impact)  +  (H  unit  O&S  Cost  * 

H  subjective  impact))  (  8  ) 


The  AFNSO  era  followed  the  same  convention  as  AFNH  to  calculate  flexible 
investment  cost.  Eras  with  three  or  four  epochs  suffered  greater  penalties.  For  example, 
the  AFNH  era  required  that  all  AF  aircraft  accommodate  design  flexibility  for  N  and  H 
requirements  while  the  AFN  era  required  the  accommodation  of  only  N  requirements.  A 
summary  of  the  flexible  investment  LCC  of  the  six  design  strategies  is  listed  in  Table  7: 
Flexible  Investment  Summary. 


Table  7:  Flexible  Investment  Summary 


Design  Strategy 

Era 

Flexible  Investment 
LCC 

Delta 

AF 

AF 

$48,236 

$0 

AFN 

AF 

$53,113 

$4,877 

AFSO 

AF 

$51,695 

$3,459 

AFH 

AF 

$50,627 

$2,391 

AFNH 

AF 

$55,533 

$7,297 

AFNSOH 

AF 

$58,787 

$10,551 

NOTE:  All  $  figures  BY13  in  millions 
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Implementation  cost  was  the  cost  of  an  era  if  a  given  era  occurred.  There  were  a 
few  key  assumptions  to  implementation  cost  which  were  driven  by  the  limitations  in  the 
CER  tool.  If  an  era  was  realized,  it  was  assumed  that  the  additional  aircraft  produced 
occurred  concurrently  with  AF  production.  For  example,  in  one  realization  of  an  era,  the 
option  to  exercise  an  era  may  happen  years  after  the  AF  baseline  has  finished  production. 
For  the  purposes  of  this  study,  an  era  is  exercised  immediately  and  both  the  AF  aircraft 
with  design  flexibility  and  the  epoch  specific  aircraft  (e.g.  AFN)  with  implemented 
capabilities  began  production  at  the  same  time  and  shared  identical  service  life  and 
retirement  dates  (see  Figure  2:  Era  Timeline  Assumption  and  Figure  3:  Possible  Era 
Realization).  If  this  were  not  the  case,  cost  penalties  associated  with  discordant  epoch 
timelines  within  the  era  would  be  required. 

Implementation  development  cost  was  equal  to  flexible  investment  development 
cost.  The  Implementation  production  and  O&S  cost,  and  ultimately  LCC,  were 
dependent  on  the  chosen  design  strategy.  In  Figure  5:  AF  Design  Strategy,  given  an  AF 
design  strategy,  or  the  decision  to  forgo  any  sort  of  design  flexibility,  implementation 
cost  was  equal  to  the  sum  of  the  separate  epoch  LCCs.  All  dollar  figures  are  in  BY  13  in 
millions. 
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Figure  5:  AF  Design  Strategy 


Equation  9  shows  the  implementation  LCC  of  AFN  given  an  AF  design  strategy 


is: 


AF  -»  AFN  Implementation  LCC  =  AF  LCC  +  N  LCC 


(9) 
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Figure  6:  AFN  Design  Strategy  summarizes  the  implementation  LCC  of  each  era 


given  an  AFN  design  strategy. 
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Figure  6:  AFN  Design  Strategy 

The  implementation  LCC  of  each  era  included  the  AFN  flexible  investment  cost. 
The  flexible  investment  cost  captured  the  cost  penalty  to  the  AF  aircraft  for 
accommodating  design  flexibility.  To  demonstrate  in  Equation  10,  the  implementation 
LCC  for  AFH  given  an  AFN  design  strategy  (AFN  ->  AFH  denotes  an  AFN  design 
strategy  and  an  AFH  era  realization)  is: 

AFN  -*  AFH  Implementation  LCC  =  AFN  Flex  Inv  LCC  +  H  LCC  (  10  ) 
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AFN  ->  AFN  was  an  aligned  scenario  where  flexibility  was  designed  for  and  the 
need  to  capitalize  on  the  flexible  design  occurred.  Implementation  LCC  for  this  was 
divided  into  development,  production,  and  O&S  cost  shown  in  Equations  11  -  13: 

AFN  ->  AFN  Implementation  Dev  Cost  =  AFN  Dev  Cost  (11) 

AFN  ->  AFN  Implementation  Prod  Cost  =  1  *  (AFN  Flex  Inv  Prod  Cost  + 

(#  of  N  A/C  *  N  unit  Prod  Cost))  (12) 

AFN  ->  AFN  Implementation  O&S  Cost  =  0.96  *  (AFN  Flex  Inv  O&S  Cost  + 

(#  of  N  A/C  *  N  unit  O&S  Cost))  (13) 

These  equations  captured  the  350  AF  aircraft  with  N  requirement  design 
flexibility  and  the  cost  to  produce  and  operate  the  additional  250  N  aircraft  at  their 
respective  costs.  In  equations  12  and  13,  a  modifier  was  applied  to  the  total  production 
and  O&S  cost.  This  modifier  represented  economies  of  scale  based  on  the  production 
and  sustainment  of  a  larger  fleet  of  aircraft.  The  modifier  was  calculated  by  averaging 
the  percent  increase  or  decrease  in  unit  production  and  O&S  cost  across  the  four  epochs 
when  the  number  of  aircraft  was  increased  from  the  baseline  350  aircraft  to  the  epoch’s 
respective  additional  aircraft  averages.  The  purpose  of  this  modifier  was  to  realize  any 
efficiencies  or  inefficiencies  associated  with  a  larger  fleet  of  aircraft  as  estimated  by  the 
CER  tool.  A  summary  of  all  the  production  cost  modifiers  are  represented  in  Table  8: 
Summary  of  Production  Cost  Modifiers. 
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Table  8:  Summary  of  Production  Cost  Modifiers 


Era 

#  of  A/C 

Production  Cost 
Modifier 

AF 

350 

1.00 

AFSO 

500 

1.17 

AFN,  AFH 

600 

1.20 

AFNSO,  AFSOH 

700 

1.24 

AFNH 

850 

1.37 

AFNSOH 

950 

1.41 

A  summary  of  all  the  O&S  cost  modifiers  are  represented  in  Table  9:  Summary  of 
O&S  Cost  Modifiers. 


Table  9:  Summary  of  O&S  Cost  Modifiers 


Era 

#  of  A/C 

O&S  Cost  Modifier 

AF 

350 

1 

AFSO 

500 

0.98 

AFN,  AFH 

600 

0.97 

AFNSO,  AFSOH 

700 

0.94 

AFNH 

850 

0.89 

AFNSOH 

950 

0.85 

Counter-intuitively,  unit  production  cost  increased  as  the  number  of  aircraft 
produced  increased.  To  investigate  this,  the  annual  rate  aircraft  production  rate  was  set  at 
48  aircraft.  As  fleet  size  increased  and  annual  aircraft  production  rate  stayed  equal,  a 
longer  production  run  would  increase  unit  production  cost.  Higher  annual  production 
rates  associated  with  larger  fleet  sizes  decreased  unit  production  cost;  however,  without 
any  additional  infonnation  on  annual  production  rates,  the  default  value  remained.  The 
production  cost  modifier  was  set  to  a  value  of  one  to  avoid  diseconomies  of  scale  in  the 
unit  production  cost  for  larger  fleet  sizes. 
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Figure  7:  AFNSOFI  Branch  Summary  summarizes  the  Implementation  LCC  given 


an  AFNSOH  design  strategy. 


Figure  7:  AFNSOH  Branch  Summary 


Like  the  smaller  design  strategies,  all  the  eras  within  the  AFNSOH  design 
strategy  included  AFNSOH  flexible  investment  cost  to  capture  the  AF  aircraft 
accommodating  flexible  design  requirements  for  N,  SO,  and  H.  Because  this  design 
strategy  accommodates  all  potential  requirements,  any  additional  epochs  realized  will 
also  incur  the  subjective  impact  penalty  associated  with  flexible  design.  For  example, 
given  AFNSOH  ->  AFNH,  all  the  AF  aircraft  suffer  penalties  for  accommodating  N,  SO, 
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and  H  flexibility,  all  the  N  aircraft  suffer  penalties  for  accommodating  SO  and  H 
flexibility,  and  all  the  H  aircraft  suffer  penalties  for  accommodating  N  and  H  flexibility. 

In  addition  to  the  subjective  impact  penalties,  the  production  and  O&S  costs  were 
modified  by  the  appropriate  values  from  Table  8:  Summary  of  Production  Cost 
Modifiers,  and  Table  9:  Summary  of  O&S  Cost  Modifiers. 

Table  10:  Design  Strategy  and  Era  Realization  Summary,  summarizes  all  LCCs  of 
the  six  design  strategies  and  eight  possible  eras. 


Table  10:  Design  Strategy  and  Era  Realization  Summary 


Design  Strategy 

Era 

AF 

AFN 

AFSO 

AFH 

AFNH 

AFNSOH 

AF 

$48,236 

$53,113 

$51,695 

$50,627 

$55,533 

$58,787 

AFN 

$85,861 

$85,460 

$89,320 

$88,252 

$89,482 

$95,003 

AFSO 

$78,828 

$83,705 

$77,772 

$81,219 

$92,565 

$88,777 

AFH 

$85,269 

$90,145 

$88,728 

$82,482 

$90,746 

$96,267 

AFNSO 

$116,453 

$116,052 

$115,397 

$118,844 

$120,074 

$123,795 

AFNH 

$122,894 

$122,493 

$126,353 

$120,107 

$118,553 

$127,458 

AFSOH 

$115,861 

$120,737 

$114,804 

$113,074 

$121,338 

$125,031 

AFNSOH 

$153,486 

$153,085 

$152,429 

$150,699 

$149,145 

$152,723 

NOTE:  All  $  figures  BY] 

13  in  millions 

For  each  design  strategy  where  the  era  occurred  (e.g.  AFN^AFN,  AFNH-> 
AFNH,  etc.),  the  lowest  FCC  was  realized  compared  to  the  other  possible  FCCs 
associated  with  that  design  strategy.  The  AFNSOH  design  strategy  was  the  exception  to 
this  pattern.  This  suggests  that  diminishing  returns  is  associated  with  design  flexibility. 
As  the  number  of  cost  penalties  due  to  a  flexible  design  increased  and  afflicted  a  larger 
fleet  of  aircraft,  the  diminishing  returns  increased  until,  the  AFNSOH’s  case,  it  incurred 
negative  value  to  design  for  flexibility. 
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The  values  in  Table  10:  Design  Strategy  and  Era  Realization  Summary,  were 
populated  in  a  decision  tree  using  Palisade’s  Precision  Tree  add-in  as  well  as  the 
probabilities  of  occurrences  listed  in  Table  3:  Era  Summary.  Based  upon  this  baseline 
information,  Precision  Tree  recommended  that  choosing  an  AF  design  strategy  would  net 
the  lowest  expected  LCC.  The  differences  in  design  strategy  expected  LCC  were 
compared  to  the  AF  design  strategy  and  are  summarized  in  Table  11:  Expected  LCC 
Differences. 


Table  11:  Expected  LCC  Differences 


Design  Strategy 

Design 

Strategy 

AF 

AFN 

AFSO 

AFH 

AFNH 

AFNSOH 

AF 

$0 

$2,027 

$1,833 

$579 

$4,781 

$8,597 

NOTE:  All  $  figures  BY] 

13  in  millions 

A  positive  value  for  any  of  the  design  strategies  other  than  AF  indicates  higher 
expected  LCC  associated  with  design  flexibility.  These  results  state  that  given  the 
assumptions  and  input  values  made  in  this  study,  design  flexibility  provides  no  expected 
return  on  investment.  Figure  8:  Design  Flexibility  Model  Visualization,  illustrates  one  of 
the  major  reasons  the  expected  LCC  differences  suggested  a  design  strategy  against 
design  flexibility  given  the  baseline  values  assumed  in  this  study.  The  AFN  era  was  used 
as  an  example. 
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Figure  8:  Design  Flexibility  Model  Visualization 

The  top  row  in  Figure  8:  Design  Flexibility  Model  Visualization,  illustrates  the 
LCC  to  procure  a  separate  AF  and  Navy  system  which  is  compared  to  the  bottom  row 
which  depicts  the  LCC  of  the  AFN  era.  The  AFN  implementation  equation  captured  the 
cost  savings  in  development  indicated  by  the  striped  box.  However,  for  production  cost 
there  were  two  factors  that  drove  the  AFN  production  cost  to  be  greater  than  the  sum  of 
the  AF  and  N  production  costs. 

1 .  Subjective  Impact  -  This  value  applied  a  cost  penalty  on  the  AF  aircraft  that 
now  had  to  be  produced  and  operated  with  design  flexibility  considerations. 

2.  Production  Cost  Modifier  -  Due  to  the  diseconomy  of  scale  indicated  by  the 
CER  tool,  a  value  of  one  was  assumed  and  there  were  no  economies  of  scale 
associated  with  production  cost. 

The  same  two  factors  that  affected  AFN  production  cost  worked  in  a  competing 
manner  for  AFN  O&S  cost.  The  same  subjective  impact  that  was  applied  to  AFN 
production  cost  was  applied  to  AFN  O&S  cost.  However,  according  to  the  default  inputs 
in  the  CER  tool,  the  larger  aircraft  fleets  realized  cost  savings,  effectively  reducing  the 
O&S  cost  modifier  to  a  value  of  less  than  one.  In  this  case,  the  O&S  cost  savings 
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associated  with  the  O&S  cost  modifier  outweighed  the  cost  penalty  of  the  subjective 
impact  resulting  in  an  the  AFN  O&S  cost  to  be  less  than  the  sum  of  the  O&S  costs  of  a 
separate  AF  and  N  systems. 

Several  sensitivity  analyses  were  completed  to  challenge  the  assumptions  made  in 
this  study  and  provided  further  insight  into  if  positive  value  exists  in  design  flexibility  for 
this  trainer  aircraft. 

Sensitivity  Analysis 

Sensitivity  analyses  allowed  a  few  of  the  major  assumptions  and  parameters  to  be 
examined. 

1 .  N,  SO,  and  H  epochs  fully  encompassed  AF  requirements 

2.  Probability  of  era  occurrence 

3.  Subjective  Impact 

4.  Production  and  O&S  Cost  Modifiers 

The  first  assumption  was  necessary  due  to  the  limited  nature  of  information. 
Subject  matter  experts  at  AFLCMC/XZE  were  available  to  provide  many  of  the  inputs 
into  the  CER  tool  to  develop  the  AF  cost  estimate;  however,  the  same  could  not  be  said 
for  N,  SO,  and  H  cost  estimates.  In  the  absence  of  specific  requirements  and  detailed 
inputs  to  the  CER  tool,  N,  SO,  and  H  requirements  were  treated  as  additions  to  AF 
requirements.  This  meant  that  N,  SO,  and  H  requirements  fully  encompassed  AF 
requirements.  This  assumption  had  a  profound  impact  on  the  results  of  the  study  which 
suggested  design  flexibility  yielded  negative  value.  Arguably,  N  requirements  could  be 
very  similar  to  AF  requirements.  However,  SO  and  H  requirements  could  be  less 
demanding  than  AF  requirements  in  terms  of  cost  to  design  and  implement  due  to  lower 
performance  requirements.  A  one-way  sensitivity  analysis  was  conducted  via  Precision 
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Tree  to  observe  how  decreasing  the  LCC  of  AFH’s  implementation  cost  would  impact 
the  recommended  decision.  The  range  of  values  captured  was  a  10%  increase  from  the 
baseline  value  of  $82,482M  and  a  25%  decrease.  The  strategy  region  is  shown  in  Figure 
9:  AFH  Implementation  Cost . 


Strategy  Region  of  Decision  Tree  'Expected  LCC' 

Expected  Value  of  Node  'Design  Strategy1  (D22) 

With  Variation  of  AFH  Implemenation  Cost  (B48) 
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Figure  9:  AFH  Implementation  Cost  One-Way  Strategy  Region 


At  LCC  values  of  ~$81,000M,  a  1.6%  decrease  in  the  baseline  AFH 
Implementation  Cost,  the  expected  value  of  choosing  an  AFH  design  strategy  over  the 
AF  design  strategy  became  more  beneficial.  This  result  suggested  that  with  cost 
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estimates  of  N,  SO,  and  H  that  do  not  encompass  AF  requirements,  it  is  easily  foreseeable 
that  a  flexible  design  option  would  result  in  LCC  savings.  Another  one-way  analysis  was 
similarly  implemented  by  changing  AFNSOH  implementation  cost.  The  strategy  region 
is  shown  in  Figure  10:  AFNSOH  Implementation  Cost . 


Figure  10:  AFNSOH  Implementation  Cost  One-Way  Strategy  Region 


Despite  a  25%  decrease  to  the  baseline  AFNSOH  Implementation  Cost,  the 
strategy  region  indicates  that  pursuing  an  AFNSOH  design  strategy  is  of  poor  value.  The 
fact  that  the  probability  of  occurrence  for  this  era  is  so  low  is  one  indicator  of  why  even  a 
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25%  decrease  to  the  baseline  value  has  very  little  impact  to  the  expected  value  of  the 
decision  tree. 

Another  subjective  parameter  in  this  study  was  the  distribution  of  probabilities 
among  the  eras.  A  one-way  sensitivity  analysis  on  any  one  of  the  probabilities  could  be 
completed;  however,  as  there  are  eight  outcomes  that  are  collectively  exhaustive, 
changing  the  probability  of  one  outcome  should  affect  the  probability  of  the  remainder  of 
outcomes.  The  outcome  probabilities  in  Precision  Tree  must  be  collectively  exhaustive 
in  order  for  a  sensitivity  analysis  to  be  conducted.  In  order  to  meet  this  constraint,  the 
chance  probabilities  were  automatically  nonnalized  via  the  model  setting  option  in 
Precision  Tree.  The  first  sensitivity  analysis  was  conducted  by  observing  how  a  change 
in  the  probability  of  AFN  affected  expected  LCC.  The  sensitivity  graph  is  shown  in 
Figure  11:  Normalized  AFN  Probability  One-Way  Sensitivity  Graph,  and  the  strategy 
region  is  show  in  Figure  12:  Nonnalized  AFN  Probability  One-Way  Strategy  Region. 
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Sensitivity  of  Decision  Tree  'Expected  LCC' 

Expected  Value  of  Node  'AFN  Design  Strategy1  (D22) 

With  Variation  of  Branch  Probability  'AFN'  of  Node  'Realization'  (E27) 
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Figure  11:  Normalized  AFN  Probability  One-Way  Sensitivity  Graph 


Figure  12:  Normalized  AFN  Probability  One-Way  Strategy  Region 
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The  sensitivity  graph  states  that  the  expected  value  of  the  decision  tree  decreases 
as  the  probability  of  AFN  increases.  The  strategy  region  recommends  the  AF  design 
strategy  for  lower  probabilities  of  AFN  and  recommends  an  AFH  design  strategy  and 
eventually  an  AFN  design  strategy  as  the  normalized  probability  for  AFN  continue  to 
increase.  This  result  is  likely  driven  by  the  first  major  assumption  observed,  the  fact  that 
N,  SO,  and  H  fully  encompass  AF  requirements. 

A  two-way  analysis  was  conducted  to  explore  the  tradespace  of  changing  the 
probability  of  AFN  and  the  production  cost  modifier  which  was  held  at  a  constant  value 
of  one  during  the  baseline  analysis  to  avoid  diseconomies  of  scale.  The  strategy  region 
for  the  two-way  analysis  is  shown  in  Figure  13:  Production  Cost  Modifier  &  Normalized 
Probability  Two-Way  Strategy  Region. 
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Strategy  Region  for  Node  'Design  Strategy' 
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Figure  13:  Production  Cost  Modifier  &  Normalized  Probability  Two-Way  Strategy 

Region 


As  AFN  probability  increased  and  the  production  cost  modifier  decreased,  it 
became  more  advantageous  to  choose  the  AFN  design  strategy  over  the  AFH  or  AF 
design  strategy.  This  same  trend  was  also  apparent  in  the  two-way  analysis  sensitivity 
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graph  shown  in  Figure  14:  Production  Cost  Modifier  &  Nonnalized  Probability  Two- 
Way  Sensitivity  Graph. 
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Figure  14:  Production  Cost  Modifier  &  Normalized  Probability  Two-Way 

Sensitivity  Graph 
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Due  to  the  normalization  of  probabilities  across  the  eight  outcomes,  the 
probabilities  shown  on  the  x-axis  in  Figures  11-14  were  the  raw  values  of  the  sensitivity 
analysis.  Table  12:  Raw  vs  Normalized  Probabilities  shows  the  normalized  values  of 
probability  for  a  better  understanding  of  how  the  actual  probability  of  AFN  occurring 
affected  expected  LCC. 


Table  12:  Raw  vs  Normalized  Probabilities 


AFN  Probabilities  (30%  baseline) 

Raw  Value 

Normalized  Value 

10% 

12.50% 

20% 

22.22% 

30% 

30.00% 

40% 

36.36% 

50% 

41.67% 

60% 

46.15% 

70% 

50.00% 

80% 

53.33% 

90% 

56.25% 

In  the  sensitivity  analysis,  Precision  Tree  incremented  the  AFN  probability  by  a 
specified  value,  normalized  all  the  probabilities  within  that  design  strategy,  and  output 
the  result.  As  the  probability  of  occurrence  increased,  the  expected  LCC  difference  for 
each  respective  era  approached  negative  values.  This  indicated  cost  savings.  According 
to  both  sensitivity  analyses,  a  production  cost  modifier  combined  with  a  high  probability 
of  era  occurrence  would  suggest  LCC  savings. 

The  final  parameter  observed  with  sensitivity  analysis  was  the  subjective  impact 
that  design  flexibility  imparted  on  the  AF  baseline.  The  strategy  region  is  shown  in 
Figure  15:  N  Subjective  Impact  One-Way  Strategy  Region. 
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Strategy  Region  of  Decision  Tree  'Expected  LCC' 

Expected  Value  of  Node  'Design  Strategy1  (D22) 

With  Variation  of  N  Subjective  Impact  (B64) 
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Figure  15:  N  Subjective  Impact  One-Way  Strategy  Region 


It’s  reasonable  to  see  that  as  the  subjective  impact  imparts  a  smaller  penalty  on 
flexible  design,  the  decision  to  choose  the  AFN  design  strategy  over  the  AF  becomes 
more  favorable.  However,  with  all  other  inputs  held  equal,  even  with  a  0%  impact,  an 
AFN  design  strategy  does  not  become  the  ideal  decision.  Another  two-way  sensitivity 
analysis  was  conducted  to  determine  how  changing  N  subjective  impact  and  AFN 
implementation  cost  would  affect  the  ideal  design  strategy.  The  strategy  region  for  this 
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analysis  is  shown  in  Figure  16:  N  Subjective  Impact  &  AFN  Implementation  Cost  Two- 
Way  Strategy  Region. 
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When  both  the  N  subjective  impact  and  AFN  Implementation  Cost  were  varied,  a 
-5.5%  decrease  in  the  baseline  AFN  Implementation  Cost  triggered  the  strategy  region 
for  this  analysis  to  recommend  an  AFN  design  strategy  over  the  AF  design  strategy.  As 
Implementation  Cost  further  decreased,  subjective  impact  was  given  the  freedom  to  adopt 
a  higher  value. 

Results 

The  analysis  suggests  that  designing  for  flexibility  can  be  more  cost  effective  than 
developing  two  separate  systems.  However,  based  upon  this  method,  the  value  of  design 
flexibility  is  largely  dependent  on  the  likelihood  that  an  era  will  occur,  the  subjective 
impact  the  design  will  have  on  the  production  and  O&S  costs  of  the  baseline  design,  and 
economies  of  scale  associated  with  larger  fleet  sizes.  Given  the  baseline  parameters,  the 
AFN  era  is  the  only  scenario  where  the  expected  LCC  differences  of  an  AF  baseline  with 
design  flexibility  would  be  favorable  over  two  separate  airframes.  The  output  value  of 
the  study  is  of  less  important  than  the  methodology  demonstrated.  Beyond  the  expected 
LCC  differences  calculated,  a  methodology  was  demonstrated  to  quantitatively  measure 
the  value  of  design  flexibility. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

This  chapter  summarizes  the  research  completed  in  this  study,  revisits  the 
investigative  questions,  and  provides  recommendations  for  future  action  and  research. 

Conclusions  of  Research 

This  research  concluded  that  a  methodology  could  be  developed  that 
quantitatively  measured  design  flexibility  and  the  expected  impact  to  LCC  based  upon 
era  realization.  The  quantitative  measurement  was  accomplished  by  using  LCC  as  a 
proxy  metric.  Realistically,  value  consists  of  many  factors  including  LCC.  Making  a 
decision  based  off  LCC  alone  would  be  foolish  if  certain  design  strategies  failed  to  meet 
the  key  perfonnance  parameters  of  the  baseline  system. 

Probability  of  occurrence,  the  subjective  impact  of  design  flexibility,  and  cost 
modifiers  associated  with  economies  of  scale  were  large  drivers  of  expected  LCC.  If  this 
method  were  implemented  again,  these  factors  should  be  well  understood  in  the  context 
of  the  relevant  study. 

Given  the  assumptions  of  the  study,  this  method  has  demonstrated  that  when 
expected  LCC  is  used  as  a  value  of  design  flexibility,  a  quantifiable  return  on  investment 
can  be  measured. 
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Investigative  Questions  Answered 

1 .  How  can  design  flexibility,  as  a  proxy  measure  for  design  flexibility,  be 
quantifiably  measured  in  the  early  stages  of  development  of  a  system? 

The  method  developed  in  this  study  was  driven  by  the  limitations  of  the  provided 
CER  tool.  Expected  LCC  has  been  demonstrated  as  a  proxy  measurement  for  design 
flexibility  in  previous  work  and  the  results  of  this  analysis  support  that  claim  [Ryan, 
Jacques  and  Colombi  2013]. 

2.  Can  we  measure  the  impact  to  expected  LCC  stemming  from  design  changes 
to  accommodate  flexibility  given  uncertain  future  requirements? 

There  a  numerous  factors  that  can  affect  the  impact  to  expected  LCC  when 
measuring  design  flexibility.  The  assumptions  made  in  this  study  were  to  aid  in  the 
absence  of  detailed  information  and  to  help  scope  the  many  possibilities  in  which  new 
requirements  could  be  levied  on  a  system.  Following  the  general  methodology  listed  in 
Table  13:  General  Methodology  to  estimate  Impact  to  Expected  LCC,  it  was  possible  to 
estimate  the  impact  to  expected  LCC  stemming  from  design  changes  to  accommodate 
flexibility  given  uncertain  future  requirements. 
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Table  13:  General  Methodology  to  estimate  Impact  to  Expected  LCC 


Step 

Cross-Reference 

Establish  appropriate  epoch  variables  using 
available  cost  estimation  method 

Table  4:  Epoch  Variables 

Assign  range  of  values  to  epoch  variables 
to  capture  uncertainty  and  model  LCC 

Table  5:  Epoch  Cost  Averages 

Assign  probability  of  occurrence  associated 
with  each  era 

Table  3:  Era  Summary 

Establish  design  strategy  and  era 
tradespace 

Figure  5:  AF  Design  Strategy 

Figure  6:  AFN  Design  Strategy 

Calculate  expected  LCC  of  each  design 
strategy 

Table  10:  Design  Strategy  and  Era 
Realization  Summary 

Note  optimal  design  strategy 

N/A 

Conduct  sensitivity  analysis  on 
assumptions  and  observe  changes  to 
optimal  design  strategy 

Figure  14:  Production  Cost  Modifier  & 
Normalized  Probability  Two-Way 

Sensitivity  Graph 

The  methodology  used  in  this  study  is  only  relevant  given  the  timeline 

assumptions  outlined  in  Figure  2:  Era  Timeline  Assumption. 

3.  Can  a  general  method  be  developed  that  can  be  applied  to  other  domains 
beyond  airframes? 

With  similar  resources,  the  methodology  in  Table  13:  General  Methodology  to 
estimate  Impact  to  Expected  LCC,  could  arguably  be  applied  to  other  domains. 
Depending  on  the  infonnation  available,  similar  assumptions  would  have  to  be 
established.  A  CER  tool  specific  for  the  applied  domain  will  be  required  to  substitute  the 
aircraft  CER  tool  used  in  this  study. 

Significance  of  Research 

The  ability  to  measure  design  flexibility  can  potentially  save  the  DOD  a 
considerable  amount  of  money  over  the  course  of  a  system’s  life  cycle.  With  a  general 
method  to  capture  the  expected  LCC  differences  by  choosing  a  system  with  design 
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flexibility  versus  procuring  two  separate  systems,  decision-makers  will  have  a  better  idea 
of  the  value  of  pursuing  a  flexible  design  option. 

Recommendations  for  Action 

To  improve  the  accuracy  of  the  result  of  this  study,  gathering  unique  requirements 
for  the  epochs  would  be  necessary  instead  of  assuming  they  fully  encompassed  AF 
requirements.  LCC  was  the  only  measure  of  value  in  this  study.  Realistically,  if  design 
flexibility  negatively  affected  an  epoch’s  aircraft  on  a  large  enough  scale,  key 
performance  parameters  could  be  at  risk.  Future  work  could  identify  certain  perfonnance 
characteristics  and  their  associated  penalties  due  to  the  separate  epoch  variables.  Once 
established,  breakpoints  in  the  design  flexibility  implemented  can  be  identified  where, 
despite  LCC  savings,  key  performance  parameters  would  no  longer  be  met.  At  this  point 
it  would  no  longer  be  in  the  best  interest  of  the  decision-maker  to  pursue  a  given  design 
strategy. 

A  uniform  distribution  was  assumed  when  the  average  cost  of  an  epoch  was 
calculated  across  a  range  of  aircraft  weight  and  avionics  weight.  A  distribution  based  off 
actual  or  parametric  data  would  help  improve  the  output  of  this  method. 

The  cost  modifiers  associated  with  economies  of  scale  were  calculated  by 
observing  changes  to  cost  as  modeled  by  the  CER  tool.  Further  research  could  be  done 
by  examining  existing  literature  on  economies  of  scale  and  applying  real  world  values  to 
this  method  which  would  further  improve  the  outputs  of  the  study. 

Another  area  of  subjectivity  to  be  addressed  is  the  subjective  impacts  associated 
with  each  era’s  requirements.  Like  the  cost  modifiers,  further  research  into  how 


61 


parameters  such  as  aircraft  weight  impacts  production  and  O&S  cost  would  greatly 
improve  the  estimates  of  this  study. 

Recommendations  for  Future  Research 

Taking  the  methodology  proposed  in  this  study  and  directly  applying  it  to  a 
different  domain  using  an  appropriate  CER  tool  would  be  valuable  to  see  if  similar  results 
could  be  replicated.  Research  that  replicates  this  methodology  across  multiple  domains 
would  lend  greater  credence  to  the  methodology  proposed  in  this  paper. 

Developing  a  specific  CER  tool  that  has  accommodations  for  quantifying  design 
flexibility  is  another  area  of  future  research.  Currently,  the  methodology  takes  a  CER 
tool  that  inherently  does  not  account  for  design  flexibility,  makes  several  assumptions, 
and  outputs  expected  LCC  differences.  If  a  CER  tool  could  be  created  that  accounts  for 
the  assumptions  of  this  study  such  as  concurrent  production,  equal  service  life,  equal 
disposal  times,  and  same  base  year  dollars,  the  estimate  for  design  flexibility  has  the 
potential  to  be  much  more  accurate. 

Summary 

Given  the  baseline  values  for  era  probability  of  occurrences,  subjective  impacts, 
and  cost  modifiers,  the  expected  LCC  differences  were  not  as  substantial  as  originally 
anticipated.  Sensitivity  analyses  revealed  how  varying  the  subjective  parameters 
impacted  expected  LCC  differences.  Although  this  research  could  not  point  out  specific 
break  points  due  to  the  notional  nature  of  many  of  the  inputs,  it  has  identified  certain 
regions  where  design  flexibility  may  no  longer  be  of  value  to  decision  makers. 

Ultimately,  this  study’s  goal  is  to  take  another  small  step  towards  the  DOD  valuable 
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resources  on  future  acquisitions  when  the  opportunity  exists  to  design  flexibility  into  a 
baseline  system. 
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